Date: Tue, 10 Dec 1996 15:07:48 GMT
Server: NCSA/1.4.2
Content-type: text/html
Last-modified: Mon, 25 Sep 1995 07:37:18 GMT
Content-length: 6002

<html>
<head>
<title>
Flawed Study of Ph.D. Production
</title>
</head>

<body bgcolor="#ffffff" text="#000070">

<h1>Massy-Goldman report alleging 50% CSE Ph.D. overproduction
to be re-issued due to flawed data</h1>

<h3>July 28, 1995</h3>

<p>
<hr>
<p>

During the first two weeks of July, a study of Ph.D. production
by William F. Massy of Stanford and Charles A. Goldman of RAND
received widespread attention, stimulated by a New York Times
article.

<p>
Both the study and the article turn out to have been badly
flawed.

<h3>
The study
</h3>

On behalf of the Computing Research Association, Jeffrey D. Ullman
from Stanford and Robert W. Ritchie from Hewlett-Packard met with
Bill Massy, the senior author of the study,
during the last week of July.  Here is Ullman's report,
edited slightly for clarity:

<blockquote>

It turns out that Bill Massy has just discovered that a critical
input parameter to his model -- the total number of doctorates
employed in the field -- was incorrect for CS.

<p>
Table 1.7 (p. 1-32 of the report) has 5,376 for this number.

<p>
Historical data from NSF, though, shows a steady increase to 19,800
for the two years prior to the year used as input data, and 5,376 for
that year.
The anomaly was due to a change in definition by those in NSF
responsible for collecting the data.

<p>
Massy has concluded that the figure
of 5,376 is inappropriate for use in his model.  He has decided
to extrapolate the
historical data to a figure of 21,000 instead.

<p>
As a result, CS now shows an employment gap of only 3.6% instead of
the published 50.3%.  This number is one of the lowest of all fields,
and suggests that the projections of 15 years ago calling for a
buildup to 1000 CS Ph.D.s/year are in remarkable agreement with
the revised Massy-Goldman estimate of current demand.

<p>
Massy says he is going to release the corrections to everyone who
received the report.  We also expect Massy and Goldman to issue a
full, revised report shortly.

</blockquote>

It's worth noting that the Massy-Goldman report was just that -- a
technical report, totally unrefereed and replete with numerous
obvious typographical errors in addition to the methodological
glitch in the case of computer science and engineering.

<h3>
The New York Times article
</h3>

The New York Times article leads one to believe that
the Massy-Goldman study was a "survey," when in fact
the study involves a first-order Markov model.
The study is perfectly clear on this point, so the
misleading statements are solely due to the New York
Times reporter.

<p>
A statement from the article such as:

<blockquote>
"The surplus of doctoral
computer science degrees currently awarded over the number of
those who get desirable jobs in their field is 50.3 percent ..."
</blockquote>

is thus doubly flawed:
<ul>
<li>
Because of flaws in the study, the 50.3% should be 3.6%.
<p>
<li>
Because of flaws in the article, the reader would conclude
that the study surveys the experiences of current graduates,
when in fact it uses a first-order Markov model to predict
the future based upon a boatload of assumptions.
</ul>
(Needless to say, a properly constructed, validated, and
parameterized model could be useful as a policy planning
vehicle.)

<h3>
What is a "desirable job?"
</h3>

The question of what constitutes "fit employment" for a Ph.D. is
an interesting one.  Here is a comment from
Forest Baskett, Vice President and Chief Technology Officer
for Silicon Graphics:

<blockquote>
"My company hires quite a number of Ph.D.s.  While it is
probably true that they don't have "jobs that really require a Ph.D."
[language from the Massy-Goldman study],
they tend to be better employees (by a lot) in those jobs than less
well educated and less well (academically) trained people we have also
hired.  These people are now moving up through our organization and
becoming the technical and business leaders of the company.  I believe
that this is a perfectly legitimate role and goal for Ph.D. production
even if it is not uppermost in the minds of professors in charge of that
production."
</blockquote>

Baskett goes on to say:

<blockquote>
"A trend I do find disturbing and counterproductive is the trend toward
postdoctoral fellows in computer science.  In my mind, opportunities
for postdoctoral fellowships allow students to further postpone the
making of hard career choices.  This potential for procrastination also
breaks the feedback loop from students to professors about the
"relevance" of their education, both in courses and research projects.
When students know that they have to actually get a career-defining job
at the end of their degree program and they know that academic jobs are
no longer easy to find, those students will, I believe, more
effectively vote with their feet about courses and research areas in a
way that better couples what's happening in our industry to what's
happening in our university departments.  We have seen this loss of
coupling happen in other areas.  Physics and life sciences come to
mind.
<p>
"We've had a fabulous 30 year history of great relations between our
computer science departments and a vibrant industry without
postdoctoral fellows gumming up the works.  One of the reasons has been
that the Ph.D.s who have gone into industry have been young and energetic
and fearless.  They have often quickly realized that they were still
doing their research.  In our fast moving industry, it also often just
happened to be product development.  This is a good model.  Let's not
break it."
</blockquote>
Baskett raises important questions concerning the <i>nature</i>
of Ph.D. education which deserve serious thought.

<p>
<hr>

<ul>
<li><!WA0><a href="http://www.cs.washington.edu/homes/lazowska/cra">Computing Research:  Driving Information
Technology and the Information Industry Forward</a>
<p>
<li><!WA1><a href="http://cra.org">Computing Research
Association</a> home page
</ul>

</body>
<address>
<hr>
<!WA2><a href="mailto:lazowska@cs.washington.edu">lazowska@cs.washington.edu</a>
</address>

</html>
